Concerning the problems of large scale change and small sample dataset in rail tread block defects, a rail tread block defects detection method based on improved Faster Region-based Convolutional Neural Network (Faster R-CNN) was proposed. Firstly, based on the basic network structure of ResNet-101, a multi-scale Feature Pyramid Network (FPN) was constructed to achieve the fusion of deep and shallow feature information in order to improve the detection accuracy of small-scale defects. Secondly, the Generalized Intersection over Union (GIoU) loss was used to solve the problem of insensitivity to the position of the predicted border caused by regression loss SmoothL1 in Faster R-CNN. Finally, a method of Region Proposal Network by Guided Anchoring (GA-RPN) was proposed to solve the problem of the imbalance of positive and negative samples in the training of the detection network due to the large redundancy of anchor points generated by Region Proposal Network (RPN). During the training process, the RSSDs dataset was expanded based on image preprocessing methods such as flipping, cropping and adding noise to solve the problem of insufficient training samples of rail tread block defects. Experimental results show that the mean Average Precision (mAP) of the rail tread block defects detection based on the proposed improved method can reach 82.466%, which is increased by 13.201 percentage points compared with Faster R-CNN, so that the rail tread block defects can be detected accurately by the proposed method.